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Motivation 

• Content discovery is becoming a challenge for players 


• Questions 

• What games to purchase? 

• Which content to download? 

• What items to purchase? 
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Daybreak's revenue-sharing program for user-created content 


% 

* 

Corsair Helmet 

L UNLOCK ^ 


$ 1.000 



Drudge Helmet 


L UNLOCK^ 
$ 1.000 



Dauntless Helmet 


L UNLOCK 

$ i.ooo 

; 

* 

Eclipse Helmet 

I v jjnlock2) 

$ 1.000 



Dolphin Ops Armor Decal 


L UNLOCK ^ 

$200 



Engineer s Union Armor Decal 

L UNLOCK [ 

$ 200 



Crenellated Wall 
Segment 

100® 


loveseat green and 
blue 

125 ® 


6 Burner Stove Grill 
w CopperHood 

100 ® 



Bed moonstone and 
water waterbed 



aquarium 


Fortified Corner 
Tower ME1 



250 ® 


100 ® 


149 ® 


s 


intricate staircase 
100 ® 



Stone Oven 


100 ® 


Infantry Gear in PlanetSide 2 


Housing Items in Landmark 












j MARKETPLACE | ,» GALLERY 

»« SEARCH BY^TEGORY V 


SOE Store 


ViOTl ► 


4 



Kerr an Furniture 
Bundle 

499 


25.000 Indicite 
Builder’!. Chest 

499 


50.000 Bluebell 
Builder's Chest 




25.000 Diamond 
Builder's Chest 

49 9*$r 


50.000 Marble 
Builder' s Chest 

499 


10.000 

Plumthistle 


T COMPETITION 


“W MY STORE 


C Balance: 0 ^ 


Enter Marketplace Search Tags 

P 


^-J 


Currently Trending 



Boat Oar 
100 



lounge chair 
50 # 



flower stained 
glass window 

200 



Log Cabin inner 
corner section 

30 ^ 


Recommended based on: 



10.000 Striped 25.000 Amethvn 25.000 Emerald 37.500 Ruby 5.000 Gold 25.000 Diamond lounge chair Wide Circular 10 Orange Modular HT 1 
Wood.. Builder s Chest Builder s Chest Builder's Chest Builder's Bag Builder s Chest 4way Comer Spotlights Single wall 

99 499 499 499 # 99 499 50 ^ 100 # 99 25 ggt 











GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 


Recommender Goals 

• Make relevant content easier to discover 

• Recommend content based on gameplay style, 
friends, and prior purchases 

• Improve conversion and monetization metrics 
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Recommender Results 

• Offline Experiments 

• 80% increase in recall rate over a top sellers list 

• Marketplace Results 

• Recommendations drive over 10% of item sales 

• Used by 20% of purchasers 

• Lifetime value of users that purchased 
recommendations is 10% higher than other purchasers 
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Types of Recommendations 

• Item Ratings 

• The recommender provides a rating for an item the 
player has not yet rated 

• Item Rankings 

• The recommender provides a list of the most relevant 
items for a player 
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Recommendation Algorithms 

• Content-Based Filtering 

• Collaborative Filtering 

• Item-to-Item 

• User-to-User 
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Collaborative Filtering 

• Rates items for a player based on the player's 

similarity to other players 

• Does not require meta-data to be maintained 

• Can use explicit and implicit data collection 

• Challenges include scalability and cold starts 
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User-Based Collaborative Filtering 


• Users 


• Items 


Similar 



Recommendation 
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Algorithm Overview 

Computing a recommendation for a user, U: 

For every other user, V 

Compute the similarity, S, between U and V 
For every item, I, rated by V 

Add V's rating for I, weighted by S to a running average of I 
Return the top rated items 
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Choosing an Algorithm 

• -low big is the item catalog? Is it curated? 

• What is the target number of users? 

• What player context will be used to provide 
item recommendations? 
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Landmark's Approach 

• User-to-user collaborative filtering 

• ot vation 


Large item catalog with limited annotations 
Rich game telemetry to alleviate cold starts 
Scales to millions of users 
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Prototyping a Recommender 

• Apache Mahout 

• Free & scalable Java machine learning library 

• Functionality 

• User-based and item-based collaborative filtering 

• Single machine and cluster implementations 

• Built-in evaluation methods 
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Getting Started with Mahout 

1. Choose what to recommend: ratings or rankings 

2 . Select a recommendation algorithm 

3. Select a similarity measure 

4. Encode your data into Mahout's format 

5. Evaluate the results 


6. Encode additional features and iterate 
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Similarity Measures 

• Item Rankings 

• Jaccard Index (Tanimoto) 

• Log Likelihood 

• Item Ratings 

• Cosine Similarity 

• Euclidean Distance 
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Mahout's Data 

• Item Associations 

User ID, Item ID 

1, 101 

1, 102 

2, 102 

2 , 103 

3 , 104 
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Format 

• Item Ratings 

User ID, Item ID, Rating 
1 , 101 , 5.0 

1 , 102 , 4.0 

2 , 102 , 2.5 

2 , 103 , 5.0 

3 , 104 , 1.0 
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Encoding Commerce Data 


SQL Query 

select u.UserlD, s.ItemID 
from Samplellsers u 
Join Sales s 

on u.UserlD = s.UserlD 
group by u.UserlD, s.ItemID 


Result Set 
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Generating Recommendations 

Building the Recommender 

model = new DataModel(new File("SalesData.csv")); 

similarity = new TanimotoSimilarity(model); 

recommender = new UserBasedRecommender(model, similarity); 


Generating a List 

recommendations = recommender. recommend(l, 6); 
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Evaluating Recommendations 

Precision computes the ratio of relevant recommendations 



Precision on the 
Landmark Dataset 

-•-User-Based (Tanimoto) 
-•-User-Based (Log Likelihood) 
-•-Itenn -Based (Tanimoto) 
-•-Item-Based (Log Likelihood) 
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Holdout Experiment 

• An experiment that excludes a single item from 
a player's list of purchases 

• Goals 

• Generate the smallest list that includes the item 

• Enable offline evaluation of different algorithms 

• Compare recommendations with rule-based approaches 
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Landmark's Holdout Results 


Recommendations significantly 
outperform a top sellers list 

80 % increase in the holdout 
Recall Rate at 6 items 


-•—Top Sellers —•—Recommendations 



# of Recommended Items 
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Integrating Additional Features 

• Landmark uses additional features 
to build item recommendations 


Features and Weights 

• Item purchased 1.0 

• Item liked 0.5 

• Item viewed 0.25 
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Encoding Additional Features 

select distinct u.UserlD, s.ItemID, 1.0 as Value 

from SampleUsers u 

join Sales s on u.UserlD = s.UserlD 

union select distinct u.UserlD, i.ItemID, O.E as Value 
from SampleUsers u 

join ItemLikes i on u.UserlD = i.UserlD 


union select distinct u.UserlD, i.ItemID, 0.25 as Value 
from SampleUsers u 

join ItemViews i on u.UserlD = i.UserlD 
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Deployment in Landmark 

• In-house implementation 

• Current Deployment 

• Recommendations are generated on the fly and cached 

• Planned Expansion 

• An offline process builds a user-similarity matrix 

• An online process generates item recommendations in 
near real-time 
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Summary 

• Recommendation systems can be applied to 
content discovery in games 

• Libraries enable rapid prototyping 

• Recommendations can significantly outperform 
rule-based approaches 
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Thank You 

• Ben G. Weber (@bgweber) 

• Director of Business Intelligence & Analytics 

• Daybreak Game Company 

• Further Reading 

• Amazon.com Recommendations: Item-to-Item 
Collaborative Filtering 

• Mahout in Action 



